{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b4da38ad",
   "metadata": {},
   "source": [
    "# 1. Libraries & Sample Data\n",
    "The first step is to load our Python Libraries and download the sample data. The dataset represents Apple stock price (1d bars) for the year 2010"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1aa05430",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load Python Libraries\n",
    "import math\n",
    "import keras\n",
    "import random\n",
    "import datetime\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "from tqdm.notebook import tqdm\n",
    "from collections import deque\n",
    "from IPython.display import display, HTML\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# for dataframe display\n",
    "pd.set_option('display.max_rows', None)\n",
    "def display_df(df):\n",
    "    # Puts the scrollbar next to the DataFrame\n",
    "    display(HTML(\"<div style='height: 200px; overflow: auto; width: fit-content'>\" + df.to_html() + \"</div>\"))\n",
    "\n",
    "# for reproducability of training rounds\n",
    "keras.utils.set_random_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa831031",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download Sample Data AAPL_2009_4m_training_features_1d (cleaned, not normalized, with features)\n",
    "data ="
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60e8526a",
   "metadata": {},
   "source": [
    "# 2. Train / Test Split\n",
    "Now that we have loaded our cleaned price dataset, we are ready to feed the data into our model. With this in mind, we select Close as our singular training feature, and split the data ito train and test data (80/20 split)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7557879",
   "metadata": {},
   "outputs": [],
   "source": [
    "# split dataset df into train (80%) and test (20%) datasets\n",
    "\n",
    "train_df =  # define training dtaframe under this variable name\n",
    "test_df =  # define testing dtaframe under this variable name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1dca49e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# display train and test dfs (ensure no overlap)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92a6ed95",
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert train and test dfs to np arrays with dtype=float\n",
    "X_train = # define training array under this variable name\n",
    "X_test = # define testing array under this variable name\n",
    "# print the shape of X_train to remind yourself how many examples and features are in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "714c77a8",
   "metadata": {},
   "source": [
    "# 3. Define the Agent\n",
    "Now that our data is ready to use, we can define the Reinforcement Learning Agent."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abd06b6d",
   "metadata": {},
   "source": [
    "### Define the DQN Model\n",
    "The first step in defining our agent is the Deep Q-Network model definition. For this project, we are creating a model sequential model with four layers. The first three layers have output shape of 64, 32, and 8, respectively, and a RELU activation. The output layer has an output shape of the size of our action space (buy, sell, hold), and a linear activation. Our Loss finction is Mean Squared Error, and our optimizer is Adam with a learning rate of 0.001. Use Keras to build this model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1e98591",
   "metadata": {},
   "outputs": [],
   "source": [
    "@keras.saving.register_keras_serializable()\n",
    "# Define DQN Model Architecture\n",
    "class DQN(keras.Model):\n",
    "    def __init__(self, state_size, action_size):\n",
    "\n",
    "        # define model layers in keras\n",
    "\n",
    "        # compile model in keras\n",
    "\n",
    "        # save model to DQN instance\n",
    "        self.model = model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edabcd7d",
   "metadata": {},
   "source": [
    "### Define Agent Class\n",
    "Now that we have defined our underlying DQN Model, we must define out Reinforcement Learning Agent. The agent initialization is provided for you, you must define an act function, and an expereince replay function. As a reminder, the act function defines how our model will act (buy, hold, or sell) given a certain state. The Experience Replay function tackles catastrophic forgetting in our training process, by maintaining a memory buffer to allow training on independent / randomized minibatches of previous states. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1489f4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Agent:\n",
    "    def __init__(self, window_size, num_features, test_mode=False, model_name=''):\n",
    "        self.window_size = window_size # How many days of historical data do we want to include in our state representation?\n",
    "        self.num_features = num_features # How many training features do we have?\n",
    "        self.state_size = window_size*num_features # State size includes number of training features per day, and number of lookback days \n",
    "        self.action_size = 3 # 0=hold, 1=buy, 2=sell\n",
    "        self.memory = deque(maxlen=1000) # Bound memory size: once the memory reaches 1000 units, the lefthand values are discarded as righthand values are added\n",
    "        self.inventory = [] # Inventory to hold trades\n",
    "        self.model_name = model_name # filename for saved model checkpoint loading\n",
    "        self.test_mode = test_mode # flag for testing (allows model load from checkpoint model_name)\n",
    "\n",
    "        self.gamma = 0.95\n",
    "        self.epsilon = 1.0\n",
    "        self.epsilon_min = 0.01\n",
    "        self.epsilon_decay = 0.995\n",
    "        \n",
    "        self.model = keras.models.load_model(model_name) if test_mode else self._model()\n",
    "\n",
    "    #Deep Q Learning (DQL) model\n",
    "    def _model(self):\n",
    "        model = DQN(self.state_size, self.action_size).model\n",
    "        return model\n",
    "    \n",
    "    # DQL Predict (with input reshaping)\n",
    "    #   Input = State\n",
    "    #   Output = Q-Table of action Q-Values\n",
    "    def get_q_values_for_state(self, state):\n",
    "        return self.model.predict(state.flatten().reshape(1, self.state_size))\n",
    "    \n",
    "\n",
    "    # DQL Fit (with input reshaping)\n",
    "    #   Input = State, Target Q-Table \n",
    "    #   Output = MSE Loss between Target Q-Table and Actual Q-Table for State\n",
    "    def fit_model(self, input_state, target_output):\n",
    "        return self.model.fit(input_state.flatten().reshape(1, self.state_size), target_output, epochs=1, verbose=0)    \n",
    "    \n",
    "    # Agent Action Selector\n",
    "    #   Input = State\n",
    "    #   Policy = epsilon-greedy (to minimize possibility of overfitting)\n",
    "    #   Intitially high epsilon = more random, epsilon decay = less random later\n",
    "    #   Output = Action (0, 1, or 2)\n",
    "    def act(self, state): \n",
    "        # Choose any action at random (Probablility = epsilon for training mode, 0% for testing mode)\n",
    "        if not self.test_mode and random.random() <= self.epsilon:\n",
    "            # **select random action here**\n",
    "        # Choose the action which has the highest Q-value (Probablitly = 1-epsilon for training mode, 100% for testing mode)\n",
    "        # **use model to select action here - i.e. use model to assign q-values to all actions in action space (buy, sell, hold)**\n",
    "        # **return the action that has the highest value from the q-value function.**\n",
    " \n",
    "    # Experience Replay (Learning Function)\n",
    "    #   Input = Batch of (state, action, next_state) tuples\n",
    "    #   Optimal Q Selection Policy = Bellman equation\n",
    "    #   Important Notes = Model fitting step is in this function (fit_model)\n",
    "    #                     Epsilon decay step is in this function\n",
    "    #   Output = Model loss from fitting step\n",
    "    def exp_replay(self, batch_size, losses):\n",
    "        losses = []\n",
    "        # define a mini-batch which holds batch_size most recent previous memory steps (i.e. states)\n",
    "        for state, action, reward, next_state, done in mini_batch:\n",
    "            # reminders: \n",
    "            #   - state is a vector containing close & MA values for the current time step\n",
    "            #   - action is an integer representing the action taken by the act function at the current time step- buy, hold, or sell\n",
    "            #   - reward represents the profit of a given action - it is either 0 (for buy, hold, and sells which loose money) or the profit in dollars (for a profitable sell)\n",
    "            #   - next_state is a vector containing close & MA values for the next time step\n",
    "            #   - done is a boolean flag representing whether or not we are in the last iteration of a training episode (i.e. True when next_state does not exist.)\n",
    "            if done:\n",
    "                # special condition for last training epoch in batch (no next_state)\n",
    "                optimal_q_for_action = reward  \n",
    "            else:\n",
    "                # target Q-value is updated using the Bellman equation: reward + gamma * max(predicted Q-value of next state)\n",
    "                optimal_q_for_action = #  reward + gamma * max(predicted Q-value of next state)\n",
    "            # Get the predicted Q-values of the current state\n",
    "            # Update the output Q table - replace the predicted Q value for action with the target Q value for action \n",
    "            # Fit the model where state is X and target_q_table is Y\n",
    "            history = self.fit_model(#X, Y)\n",
    "            losses += history.history['loss']\n",
    "           \n",
    "        # define epsilon decay (for the act function)\n",
    "        if self.epsilon > self.epsilon_min:\n",
    "            self.epsilon *= self.epsilon_decay\n",
    "        return losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63b8c216",
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.utils.disable_interactive_logging()\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "window_size = 1\n",
    "agent = Agent(window_size, num_features=X_train.shape[1])\n",
    "dot = keras.utils.model_to_dot(\n",
    "    agent.model,\n",
    "    show_shapes=True,\n",
    "    show_dtype=True,\n",
    "    show_layer_names=True,\n",
    ")\n",
    "dot.write(\"model.png\", format='png')\n",
    "from IPython import display\n",
    "\n",
    "display.Image('model.png')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "rl-project-udacity",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.1.undefined"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
